This application is based upon and claims the benefit of priority from the prior Japanese Patent Applications No. 11-296686 filed Oct. 19, 1999; No. 11-305565 filed Oct. 27, 1999; and No. 11-307079 filed Oct. 28, 1999, the entire contents of which are incorporated herein by reference.
The present invention relates to an information display member such as an information display board on which information containing a position of an object is displayed, and to a position detecting method using the display member.
In addition, the present invention relates to a related information presenting apparatus and a related information presenting method for displaying information associated with an object in an image.
Further, the present invention relates to an information presenting apparatus and an information presenting method, and in particular to an information presenting apparatus and an information presenting method for presenting a user with information associated with an object of the user""s interest.
For example, according to Jpn. Pat. Appln. KOKAI Publication No. 6-197246, there is disclosed that a two-dimensional bar code or the like is prescribed on a road sign, whereby information is automatically read by means of a bar code reader device comprising a CCD camera without a human being reading such road sign.
In addition, according to Jpn. Pat. Appln. KOKAI Publication No. 10-267671, there is disclosed associated information display technique as shown in FIG. 1 for the purpose of displaying information associated with an image from image input means.
In the associated information display technique, the following is described. That is, an approximate position during image acquisition caused by an image acquisition section (image acquisition apparatus) 51 is acquired from a position information acquisition section 52, and an angle is acquired by means of a camera attribute information acquisition section 53. Further, a view image of field of view that can be viewed from that position is created by using a computer graphics (CG) technique by means of a control section 58A based on three-dimensional map information from a map information management section 55. The thus created image is superimposed with label information from a label information creating section 56A by means of a label information output section 57A, whereby an input image and pattern matching are obtained.
However, in the aforementioned Japanese Patent Application Laid-open No. 6-197246 that is prior art, in a method of using a two-dimensional bar code, human readable language or sign information that an attached material of that code displays is merely replaced with code information in accordance with another standard such as existing bar code, is merely easily readable in view of image processing, and is not largely different from the identity of an inherent bar code.
That is, precise map information or position information on where it is or where a car is bound for is required for human navigation or navigation while driving a car. In this respect, the conventional information display and position and orientation recognizing method are not satisfactory.
In addition, in the associated information display technique disclosed in Jpn. Pat. Appln. KOKAI Publication No. 10-267671 that is prior art, it is required to reconstruct a view image in the field of view and obtain matching from a three-dimensional map containing information on height direction in order to determine associated information on superimposition.
Therefore, in this associated information display technique, there is a problem that much calculation time is required for the matching process, and it is difficult to identify a precise camera position (user position).
In the meantime, when an attempt is made to know a self position or direction, one always attempts to acquire visual information on periphery.
Nobody suspects this fact as long as he or she is an ordinary person.
When one moves to a land that he or she does not know well, one would attempt to know where he or she is based on tools such as a map or a compass or an astronomical position.
However, in the case where such process is performed automatically by means of an instrument in an image processing manner, non-visible position detecting means (such as GPS or gyrocompass) is avoidably used.
In addition, it is known that a method of obtaining a position and orientation of image input means from a reference mark position is disclosed in reference 1 xe2x80x9cA High Accuracy Realtime 3D Measuring Method of Marker for VR Interface by Monocular Visionxe2x80x9d (3D Image Conference"" 96 Collection of Papers, pp. 167-172, Akira Takahashi, Ikuo Ishi, Hideo Makino, Makoto Nagasizu, 1996).
FIG. 2 shows a method of obtaining a position and orientation of the image input means disclosed in reference 1.
That is, in this method, in FIG. 2, when Oc is defined as a viewing point, xe2x80x98poxe2x80x99 is defined as a cross point of diagonal line of a projected image, and xe2x80x98Poxe2x80x99 is defined as a cross point of diagonal line for a rectangle, the orientation is obtained by obtaining xxe2x80x2 such that the following equation is minimized.       F    ⁢          (              x        xe2x80x2            )        =      4    -                  ∑                  i          =          1                4            ⁢                        (                                    n              i                        ⁢                          n              i              xe2x80x2                                )                2            
Here, a variable vector xxe2x80x2 is xxe2x80x2=(xcex11xe2x80x2 xcex21xe2x80x2, xcex12xe2x80x2).
With respect to a position, when xe2x80x9cdxe2x80x9d is estimated such that a distance from Ocpi and apex Pi is minimized by the minimum square method, the position is obtained by the formula below.   d  =                                          (                                          S                o                            ⁢                              S                i                                      )                    ⁢                      xe2x80x83                    ⁢                      (                                          S                i                            ⁢                              W                i                                      )                          -                  (                                    S              o                        ⁢                          W              i                                )                            1        -                              (                                          S                o                            ⁢                              S                i                                      )                    2                      ⁢          xe2x80x83        ⁢    D  
Then, a position can be defined by obtaining an average of depth values obtained from each apex.
In addition, in reference 2 xe2x80x9cMixed Reality (ISBN 4-274-90283-8 Ohmsha, Ltd., Chapter 6: Augmented Reality Tracking in Natural Feature Environments, U. Neumann, S. You, Y. Cho, J. Lee and J. Park 1999 pp. 101-130), there is disclosed a technique of obtaining an initial position of image input means by using a known reference point called Fiducial Mark.
In this reference 2, a plurality of feature points are extracted, a projection position in an input image is obtained, and the feature points are tracked according to movement of the input device, whereby an optimal solution of the position of the image input means is obtained.
However, as disclosed in the aforementioned paper (reference 1) authored by Takahashi et al, the position and orientation of the image input means can be detected by the position coordinate of four corners of the reference mark itself. In the case where a distance between a camera and a mark is distance from each other, there is a problem that the projection distance in the field of view of four corners is reduced, and the detection precision is impaired.
In addition, using a large mark is often impractical.
In the aforementioned prior art disclosed in reference 2, it has been required to additionally record the known coordinate of a reference point or to provide a reference table between the reference point and the coordinate in order to obtain an initial position of the image input means.
Hence, in the case where the prior art disclosed in reference 2 is used in an area with its wide range or in the case where the prior art is used in a small terminal, there is a problem that a limited number of the coordinate can be recorded, and it takes long to obtain matching with the current extracted reference points from among a number of reference points, and thus, the prior art is used in a limited region, which lacks general purpose properties.
Conventionally, information presenting apparatuses for presenting information concerning on object of the user""s interest in the real world for a user who is active in the real world include a portable information presenting apparatus composed of a liquid crystal display for presenting information to the user and a small video camera for acquiring peripheral images (Masato Hirakawa, Hisaaki Yasumura xe2x80x9cVisual Interfacexe2x80x9d, bit special issue, Kyoritsu Shuppan, pp. 177-198, 1996).
In this information presenting apparatus, useful information concerning an object that exists at the periphery is associated with a position at which an object exists, and is presented to user in the real world.
In this information presenting apparatus, an ID recognition color code (a kind of bar code in which ID is encoded by red and blue stripes) is appended by each object, and an image of an object including a color code acquired from a camera is transmitted to an image recognition section.
In this image recognition section, identification is performed on an object of the user""s interest by decoding an ID number represented by a color code, and the identified object is transmitted to a message generating section.
In this message generating section, an identified ID is defined as a search key, and the corresponding information is searched from a database provided in advance. Then, a message generated based on this searched information is composed by means of an image composing section at a proper position on an image acquired from the camera, and the message is displayed on an information presenting display.
As an example of using such information presenting apparatus, when the user applies this apparatus in front of a calendar hanging on a wall, schedule information acquired from the database is considered to be composed and displayed on the calendar image.
In the aforementioned information presenting apparatus based on the prior art, an identification color code mounted to each object is identified, and an identification ID is defined as a search key. Then, the corresponding information is searched from the provided database, and the information to be composed and displayed is generated.
Therefore, in the aforementioned information presenting apparatus based on the prior art, it is required to provide an identification ID to each object. Further, there is a problem that information cannot be presented as long as a database is not provided by each identification ID.
It is a first object of the present invention to provide an information display member employed when an attempt is made to know the self position and orientation or camera position and orientation using at least visual information, in other words, image information.
It is a second object of the present invention to provide a position detecting method in which a self position is conventionally determined in a known triangulation manner by employing this information display member, thus making it possible to eliminate a need to input the position coordinate of two points or the direction or orientation of the presence of the known landmark.
It is a third object of the present invention to provide an associated information presenting apparatus capable of determining associated information to be superimposed at a high speed and capable of obtaining its coordinate position by detecting a reference point.
It is a fourth object of the present invention to provide an associated information presenting method capable of determining associated information to be superimposed at a high speed and capable of obtaining its coordinate position by detecting a reference point.
It is a fifth object of the present invention to provide an information presenting apparatus capable of reliably presenting target information to a user even if there is no need to provide an identification ID by each object, and further, even if there is no database for storing all information to be displayed.
It is a sixth object of the present invention to provide an information presenting method capable of reliably presenting target information to a user even if there is no need to provide an identification ID by each object, and further, even if there is no database for storing all information to be displayed.
In order to achieve the foregoing first object, according to a first aspect of the present invention, there is provided an information display member having an information display face of its known size, the information display member comprising:
a first pattern displayed on the information display face in order to pattern and represent coordinate information indicative of a position of the information display member; and
a second pattern displayed on the information display face in order to pattern and represent direction information indicative of a direction oriented on the information display face.
An information display face of the information display member is acquired by means of a camera, whereby the camera position information is acquired by the acquired image information based on a first pattern representing coordinate information indicative of a position of the information display member displayed on the information display face; and a second pattern representing direction information indicative of a direction oriented on the information display face.
On the information display face, there may be further represented a third pattern representing height information indicative of height of the information display member and a fourth pattern in which vertical and horizontal directions are represented to be patterned.
An information display face of the information display member is acquired by means of a camera, whereby camera position information is acquired from only acquired image information, and height information on the camera position is acquired based on: a first pattern presenting coordinate information indicative of a position of the information display member displayed on the information display face; a second pattern representing direction information indicative of direction oriented on the information display face; a third pattern representing height information indicative of height of the information display member; and a fourth pattern representing the vertical and horizontal directions.
In order to achieve the aforementioned second object, according to a second aspect of the present invention, there is provided a position detecting method, comprising the steps of:
acquiring an image by acquiring the information display member displayed on the information display face with a camera by a first pattern for patterning and representing coordinate information indicative of a position an information display member having an information display face of its known size and a second pattern for patterning and representing direction information indicative of a direction oriented on the information display face;
obtaining a length ratio of aspect on the information display face mapped on the image and a direction in which the camera is oriented from direction information displayed on the information display face;
obtaining a distance between the camera and the information display face from the size of the information display face mapped on the image; and
obtaining the position coordinate of the camera from the direction in which the camera is oriented, the distance, and the coordinate information displayed on the information display face.
In the case of the above information display member, there are further displayed on the information display face a third pattern for patterning and representing height information indicative of height of the information display member; and a fourth pattern for patterning and representing the vertical and horizontal directions of the information display member, wherein the position detecting method may further comprise the steps of: obtaining height information displayed on an information display face mapped on the image; and obtaining the height of the camera from the height information and an elevation angle of the camera.
In order to achieve the third object, according to a third aspect of the present invention, there is provided an associated information presenting apparatus for superimposing and displaying the associated information on each section of the object at the corresponding position on an image on which an object is image-acquired, the associated information presenting apparatus comprising:
an image input section for including in the range of image acquisition a mark on which a pattern representing position information on four points or more of the object is displayed, and inputting an image in which the object is image-acquired;
a position information management section for reading the pattern from an image inputted by the image input section, and acquiring the position information;
a position information acquisition section for obtaining the position and orientation of the image input section from acquired position information acquired by the position information management section;
an associated information generating section for obtaining the position of each section of the object on the image based on the position and orientation of the image input section obtained by the position information acquisition section; and
an information superimposing and displaying section for superimposing and displaying the associated information at the corresponding position based on a position on the image obtained by the associated information generating section.
In order to achieve the fourth object, according to a fourth aspect of the present invention, there is provided a method for superimposing and displaying associated information on each section of the object at the corresponding position on an image on which an object is image-acquired, the method comprising the steps of:
including in the range of image acquisition a mark on which a pattern representing position information on four points or more of the object is displayed, and inputting an image on which the object is image-acquired by an image input section;
reading the patter, thereby acquiring the position information;
obtaining the position and orientation of the image input section from the position information;
obtaining the position of each section of the object on the image based on the position and orientation of the image input section; and
superimposing and displaying the associated information at the corresponding position based on the position on the image.
In order to achieve the foregoing object, according to a fifth aspect of the present invention, there is provided an information presenting apparatus, comprising:
an image input section for inputting an image;
a pattern recognition section for recognizing a pattern representing information on an image inputted by the image input section;
an information reproducing section for representing information represented by a pattern recognized by the pattern recognition section;
an image superimposing section for superimposing information reproduced by the information reproducing section on an image inputted by the image input section; and
an image display section for displaying an output from the image superimposing section.
In order to achieve the sixth object, according to a sixth aspect of the present invention, there is provided an information presenting method comprising the steps of:
inputting an image;
recognizing a pattern representing information on an input image;
reproducing information representing a recognized pattern;
superimposing reproduced information on the input image; and
displaying an image on which the reproduced information is superimposed.
Additional objects and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objects and advantages of the invention may be realized and obtained by means of the instrumentalities and combinations particularly pointed out hereinafter.